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Abstract 

Using an asymmetric associative network with synchronous updating, it 
is possible to recall a sequence of patterns. To obtain a stable sequence 
generation with a large storage capacity, we introduce a threshold that 
eliminates the contribution of weakly correlated patterns. For this sys- 
tem we find a set of evolution equations for the overlaps of the states 
with the patterns to be recognized. We solve these equations in the 
limit of the stationary cycle, and obtain the critical value of the capac- 
ity as a function of the threshold and temperature. Finally, a numerical 
simulation is made, confirming the theoretical results. 

* This work is supported in part by CONACyT grant 400349-5-1714E and 
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1. Introduction 

The application of neural networks to time-sequence recognition has a wide 
range of potential applications, from associative memory to speech recognition 
to the prediction and control of complex dynamical systems. Networks with 
symmetric synapses, such as the Hopfield and Little models are unable 
to reproduce complex sequences since they minimize an energy function t^'^l. 
The equilibrium properties of these models have been studied analytically by 
constructing the partition function associated to the energy. In Ref. [5] the asyn- 
chronous case (the Hopfield model) and the synchronous one (the Little model) 
have been studied far from saturation. That is to say, for a = with a = p/N, 
where p is the number of patterns and N is the number of neurons as N goes 
to infinity. The near saturation case {a ^ 0) was studied by Amit et al. for the 
asynchronous case and by Fontanari et al. for the synchronous one by us- 
ing replica symmetry techniques. Networks with asymmetric synapses are able 
to reproduce sequences of patterns, however in general the equilibrium statisti- 
cal mechanics cannot be applied. Instead of using an energy approach Coolen 
and Ruijgrok started from the master equation for the microscopic states 
of the network and derived an evolution equation for the probability density 
for the overlaps of the states with the patterns in an asynchronous asymmetric 
network. Using this approach other authors have studied asynchronous asym- 
metric nets far from saturation t^'^o]. The synchronous case was also analyzed 
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far from saturation by Bernier I^^L 

Near saturation networks have been studied recently with success using 
statistical methods based on the central limit theorem (CLT). In Ref. [12] Nishi- 
mori and Ozeki investigate the relaxation to equilibrium in the Little model 
(synchronous dynamics) near saturation using a mean field treatment and solv- 
ing the equations by means of the CLT. They obtained in the equilibrium limit a 
good qualitative agreement with the results by Amit Gutfreund and Sompolin- 
sky . An energy approach and the CLT was used by Shukla ^^^^ in the Little 
model to obtain a phase diagram that coincide well with the numerical results. 
In Ref. [14] we studied a symmetric, asynchronous, state-dependent network 
with a threshold using a CLT method developed by Geszti and Peretto t^^^ and 
find that its capacity increases with 77 (the threshold) as a ~ 77"^ 6^^/^. More- 
over for ?7 = we recovered the Hopfield model an found the same results as 
with replica symmetry techniques t^^. 

In this article, we will consider an asymmetric neural network 
synchronously updated with a state-dependent synapses and calculate its ca- 
pacity to follow a sequence of patterns. For this we first derive the evolution 
equations for the overlaps using the transition probability for the neuron states 
and a mean field type approximation. We then solve them by the statistical 
method of Geszti and Peretto. We will show that this network has a maximal 
storage capacity ckc — p/N — 0.278 in the case rj — 0, which is the limit in which 
the network is state-independent. Beyond this value, the number of weakly cor- 



related patterns becomes so large that their contribution dominates over that 
of the highly correlated pattern, which should force the transition to the next 
state in the sequence, and the system diverges from the learned sequence. We 
will show how the storage capacity increases if one limits the contribution of 
weakly correlated patterns, by taking rj 0: Only those patterns whose cor- 
relation with the state of the system is greater or equal to the threshold are 
left to give a contribution to the synapses. We will give the dynamical phase 
diagram of this network and compute the critical capacity as a function of the 
temperature and the threshold: the learned sequence is stable up to the critical 
capacity, which increases with the threshold. These results are confirmed in the 
deterministic limit T = 0, with the help of a numerical simulation. 

Although biological synapses are likely to be asymmetric, and this fact is 
probably an important factor in the complexity observed in biological neural 
systems, the application of a state-dependent threshold and the use of a central 
clock for the parallel updating of the neurons implies that the results of this 
article are not likely to be relevant in the physiological domain. On the other 
hand, the use of synchronous parallel updating allows for an efficient use of 
modern parallel-processing computers. We will discuss possible applications of 
our work to the prediction of chaotic time series in the conclusion. 

In Sec. 2, we review the increase in capacity with the threshold parame- 
ter in the Hopfield model, as well as the solutions of the mean field equations 
for symmetric synapses In Sec. 3, we consider the asymmetric rule with 
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a threshold, and paraUel updating of the neurons; the evolution equations for 
the overlaps are derived and solved using the statistical method developed in 
Ref. [15], and the generalized phase diagram for the stationary limit of the dy- 
namics is constructed. In Sec. 4, we give the results of the numerical simulation 
and in Sec. 5 we give the conclusions. 



2. State-Dependent Synapses and Associative Memory. 



In Ref. [14] we considered a network with symmetric state-dependent 
synapses for associative memory tasks. The main idea of this type of synapses 
is to cut off the contribution of those patterns whose correlation is less than a 
given value (the threshold). A set of N neurons with states sf = ±1 at time 
n = 1,2,... (i = 1,...,N) interact through a state-dependent synapses matrix 
Wlj. The network evolves by updating one neuron at a time in random order 
through the rule 

s^+i = ±1 with probability ^^oah-^ (^) 
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where 
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and where (3~^ = T is the temperature parameter. Let {^f = ±1} (/i = 1, 
be a set of p patterns generated randomly, such that 

«erej»= (3) 

where <^ ... 3> denotes the average over the distribution of patterns. The 
synapses matrix was taken to be 

^5 = ^Ec%^ -7^)- (4) 

where 



is the correlation of the state with the pattern and the step function (x) 
ensures that the pattern is set to zero if (m^ (s"'))^ < rf /N. 77 is called the 
threshold and for ry = Hebb's rule for the synapses is recovered I^'^^l. From 
(1) and (5) one obtains the equation of motion for the instantaneous average 
activity of the neurons 



where 
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I ^<m,is-)>= ^Ee;<^;>, (7) 
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< ... > denoting the thermal average over the distribution (1). Since 
m'^ — (s") ~ O (^"^ j can substitute (s") by in (6) making 
an error of 
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where a = p/N and Prob [x > 0] is the probability that x be greater or equal 
to zero. In the limit a — > this substitution is exact, while for o: the 
approximation remains good for large values of rj since then the probability 
that a weakly correlated pattern pass the threshold becomes small. With this 
approximation, one obtains the mean field equations 

= ^ E t^^h |/3 J] ejm-e (^{m:f - ^) I . (9) 

A study of these equations by means of the statistical techniques developed in 
Ref. [15] showed that the storage capacity etc = p/N can be increased up to 



any preassigned value (for — > oo) by taking a sufficiently large value of the 
threshold. Those results were confirmed by means of a numerical simulation. 

3. Sequence Recognition with State-Dependent Synapses 

Let us consider a neural network with N neurons and a time series of q 
random patterns {^f } which obeys (3). We want the network to be capable of 
following the sequence of patterns in order from = 1 to /i = p, where p < q. 
The capacity of the network is again defined by a — p/N. The system evolves 
now in parallel by the transition probability from the state s' = (sQ to the state 



The evolution of the probability of the network P„ (s) to be in state s at time 



0(s'^s) = n ^[l + s,tanh(/?/iO] 



(10) 



i=l 

where /i^ is given by (2) and the synapses is now 
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n is given by 1^°'^^] 




(12) 
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The average overlap at step n + 1 is then 




(13) 



where 

(^)„^5]P„(s)^(s). (14) 
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After substituting (10) and (12) in the expression (13) and then summing over 
s, one finds 

We can substitute the overlaps in the right hand side of (15) by its thermal 
average making an error of order given by (8), obtaining 



We are interested in finding solutions of (16) which have a large overlap only 
with the pattern fj, = n: 

«m-»=5;m„. (17) 

Let us consider in (16) fx = i' + l^n+l and expand the term proportional to 
m^Q (^(m^)^ — rj'^/N^ to first order. One finds 

<tl = ^ E^r'^r^' tanh [P {run + T^'^)] 

i 

+ ^r^Q ({rKf - ^) E [1 - t^^h^ (™" + '^r'^)] (18) 

where 

^ E ^t'^i^'K © iiKf - • (19) 

Because the m^, jJi ^ n are the sum of a large number of random variables 
< > we may assume that they have normal distributions centered at 



zero, with variance o'^jN ^ by the CLT. For 7^ 0, only the patterns which pass 
the threshold contribute to (19): the distribution of over the reduced set 
of patterns is gaussian for (mj];)^ > rf" jN and zero in the strip (mj])^ < rf /N . 
We will assume that the number of patterns which pass the threshold is large, 
so that T^'^ has a normal distribution also for ry ^ 0, with variance ar^ and 
average zero (note, however, that the m^, A* 7^ not strictly independent 

variables, since they are related through the equations (16)). Then 



avn = « iV'^f », (20) 

and ^ 

^^^^^ ^ + ^i'"'') 

r dz . ^^^^ 

= / e-^'/2 tanh^ (3 {rrin + ^foa^z) • 

From (18) we get 

mill - «G Uml) - ^) (1 - ?n) 

1 _ (22) 
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Squaring this expression, averaging over the distribution of patterns and using 
(19), (20) and (22) we get for p > 1 

cr^+i (1 - gn)^r^ = 9n (23) 

Another relation can be obtained directly from equations (19) and (20), giving 

r„ = -|.Jr(3/2,^) (24) 
1 n 



where T is the incomplete gamma function t^^L Taking now the mean field 
equations for = n + 1 we obtain, with similar calculations, 



Equations (21), (23), (24) and (25) are simultaneous equations for the 
unknowns m„, g„, cr^ and Vn- We can solve them in the stationary limit in 
which the order parameters are constant in time: = m, qn = q, CTn = and 
Vn = r. In [Fig. 1], we describe the space of solutions at T = 0: for 77 = there 
exist solutions for a < 0.278, while for ry this value increases as is shown 
by the critical line adrj); m 7^ 0. Above this line one has temporal sequence 
solutions of the form (11), while below it m = 0. For 77 — > 00, ckc — ^ 00 as 



In [Fig. 2] and [Fig. 3] we give the critical capacity as a function of the 
threshold for finite temperatures. 

4. Numerical simulation 

A numerical simulation of a network of N =144 neurons was carried out. 
The network was initiated near the first pattern in each set, with an error in one 
of the neurons, and allowed to evolve according to the deterministic synchronous 




(25) 
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evolution rule with the synapses (11). The overlap with the last pattern in the 
sequence was determined, and this result was averaged over 200 sets of randomly- 
generated patterns and over 25 different choices of the erroneous initial neuron. 
For low values of the threshold the overlap begins to decline sharply at a critical 
value of the capacity close to the theoretical estimate. For t] > 2 the decline 
is smoother and it is difficult to determine accurately the critical capacity. 
Following the first referee's suggestion, we increased N to 1681 neurons, using 
the "trick" described by Penna and Oliveira ^^^1 (1681 is the square of 41; the 
odd number avoids memory drift conflicts in the vectorized code). The low-r; 
graphs were unchanged, but the new rj = 2 graph displays a sharp drop near 
the theoretical critical capacity estimate, namely at etc ~ 1.1 [Fig. 5]. 

The need for a greater number of neurons for large-?7 simulations reflects 
the fact that the number of patterns which pass the threshold must be large, 
in order to apply the CLT in (16). Since has a normal distribution with 
variance ^ j^, this number is of the order aN (l — erf [r]/y/2j^, where erf() 
is the error function For N = 144, rj = 2 and a = 1, an average of only 
6.55 patterns pass the threshold, while for A?" = 1681 this number increases to 
76.5, enough to invoke the CLT. As this interpretation suggests, there should 
be no need to increase further the number of neurons; this fact was conflrmed 
by a run ai N = 6561, which reproduced the same graph as Fig. 5. 



5. Conclusions 

We have proposed a neural network with state-dependent synapses and 
synchronous updating capable of storing long sequences of random patterns. 
In order to achieve this result, we introduced a threshold which cuts off the 
contribution of the weakly correlated patterns. The critical storage capacity 
was computed as a function of the threshold parameter and the temperature, 
and the results for the deterministic case T = were confirmed by numerical 
simulation. 

One of the motivations for this work was the possibility of applying this 
type of network to chaotic time series prediction. There it is necessary to 
"teach" the network a large number of patterns, typically of the order of 10^ 
for a low-dimensional attractor. Since the present network is capable of storing 
such a chaotic time series, it is possible that as a dynamical system this network 
would itself display a chaotic behavior, with an attractor which approximates 
reasonably well the attractor of the physical system that generated the time 
series. Finally, it may be possible to adjust the value of the threshold so that 
the first Liapunov coefficient of the neural network be equal to that of the time 
series; this would probably give a subcritical value of the threshold, such that 
the learned sequence is unstable. This approach to nonlinear modeling sug- 
gests the exciting prospect of designing models which come close to topological 
equivalence with the physical system. 
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Figure captions 



Figure 1. The increase in the storage capacity of the neural network with the 
threshold is represented, in the deterministic limit T = 0. The critical capacity 
with threshold equal to zero is etc = 0.278. With a threshold 77 = 1 the critical 
capacity increases to 0.36, and at = 2 it reaches etc = 1.1 

Figure 2. The maximum storage capacity is given as a function of the threshold 
at T = 0.4. For low values of the threshold the capacity is less than in the 
deterministic case T = 0, as thermal fluctuations inhibit a clean recall of the 
sequence. However, the improvement in the storage capacity with the threshold 
is stronger than at zero temperature, and for thresholds above rj — 1.15 the 
capacity of the T = 0.4 network surpasses that of the deterministic network. 

Figure 3. The storage capacity is given as a function of the threshold at 
T = 0.7. 

Figure 4. The results of the numerical simulation of the network is represented. 
The average overlap with the learned sequence is given as a function of the 
storage capacity a = p/N, for various values of the threshold. A sharp decline 
in the overlap is detected near the theoretical value etc = 0.28 for rj — 0. The 
increase in capacity from 77 = to 77 = 1 is roughly equal to 0.06, again close to 
the theoretical prediction. For r; = 2 the capacity increases substantially: one 
has an accurate recall of the sequence up to a = 0.6, and the overlap with the 
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sequence goes to zero near the theoretical critical value ckc = 1-1 • 

Figure 5. The numerical simulation with N — 1681 neurons reveals a critical 
capacity etc ~ 1.1, in agreement with the theoretical calculation. 
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